zookeeper 刪除snapshot和transaction log的源碼解讀
- 2020 年 3 月 5 日
- 筆記
轉載請註明源地址http://www.cnblogs.com/dongxiao-yang/p/4910059.html
zookeeper具有自動清除快照日誌和事務日誌的工能，可以在配置文件設置autopurge.purgeInterval來實現，問題是這個屬性的時間單位是小時，
有些情況下，一小時的日誌過大（比如把事務日誌放到內存），需要手動刪除，所以需要研究下zk刪除日誌文件的源碼。
清理日誌主類:org.apache.zookeeper.server.PurgeTxnLog,包含如下幾個靜態工具方法
static void printUsage(){
System.out.println("PurgeTxnLog dataLogDir [snapDir] -n count");
System.out.println("tdataLogDir — path to the txn log directory");
System.out.println("tsnapDir — path to the snapshot directory");
System.out.println("tcount — the number of old snaps/logs you want to keep");
System.exit(1);
}
常見的幫助方法，告訴使用者參數的傳入順序，其中snapdir參數為可選，假如兩種日誌配置在同一個路徑下，只傳一個路徑參數就好。
main方法，沒什麼好說的，只是解析參數。
public static void purge(File dataDir, File snapDir, int num) throws IOException {
if (num < 3) {
throw new IllegalArgumentException("count should be greater than 3");
}
FileTxnSnapLog txnLog = new FileTxnSnapLog(dataDir, snapDir);
List<File> snaps = txnLog.findNRecentSnapshots(num);
retainNRecentSnapshots(txnLog, snaps);
}
刪除文件的主方法，主要分兩個部分
1：txnLog.findNRecentSnapshots(num);
找到需要保留的文件
主要邏輯代碼為
public List<File> findNRecentSnapshots(int n) throws IOException {
List<File> files = Util.sortDataDir(snapDir.listFiles(), "snapshot", false);
int i = 0;
List<File> list = new ArrayList<File>();
for (File f: files) {
if (i==n)
break;
i++;
list.add(f);
}
return list;
}
private static class DataDirFileComparator
implements Comparator<File>, Serializable
{
private static final long serialVersionUID = -2648639884525140318L;
private String prefix;
private boolean ascending;
public DataDirFileComparator(String prefix, boolean ascending) {
this.prefix = prefix;
this.ascending = ascending;
}
public int compare(File o1, File o2) {
long z1 = Util.getZxidFromName(o1.getName(), prefix);
long z2 = Util.getZxidFromName(o2.getName(), prefix);
int result = z1 < z2 ? -1 : (z1 > z2 ? 1 : 0);
return ascending ? result : -result;
}
}
/**
* Sort the list of files. Recency as determined by the version component
* of the file name.
*
* @param files array of files
* @param prefix files not matching this prefix are assumed to have a
* version = -1)
* @param ascending true sorted in ascending order, false results in
* descending order
* @return sorted input files
*/
public static List<File> sortDataDir(File[] files, String prefix, boolean ascending)
{
if(files==null)
return new ArrayList<File>(0);
List<File> filelist = Arrays.asList(files);
Collections.sort(filelist, new DataDirFileComparator(prefix, ascending));
return filelist;
}
2 刪除文件
// VisibleForTesting
static void retainNRecentSnapshots(FileTxnSnapLog txnLog, List<File> snaps) {
// found any valid recent snapshots?
if (snaps.size() == 0)
return;
File snapShot = snaps.get(snaps.size() -1);
int ii=snaps.size() -1;
System.out.println(ii);
final long leastZxidToBeRetain = Util.getZxidFromName(
snapShot.getName(), PREFIX_SNAPSHOT);
class MyFileFilter implements FileFilter{
private final String prefix;
MyFileFilter(String prefix){
this.prefix=prefix;
}
public boolean accept(File f){
if(!f.getName().startsWith(prefix + "."))
return false;
long fZxid = Util.getZxidFromName(f.getName(), prefix);
if (fZxid >= leastZxidToBeRetain) {
return false;
}
return true;
}
}
// add all non-excluded log files
List<File> files = new ArrayList<File>(Arrays.asList(txnLog
.getDataDir().listFiles(new MyFileFilter(PREFIX_LOG))));
// add all non-excluded snapshot files to the deletion list
files.addAll(Arrays.asList(txnLog.getSnapDir().listFiles(
new MyFileFilter(PREFIX_SNAPSHOT))));
// remove the old files
for(File f: files)
{
System.out.println("Removing file: "+
DateFormat.getDateTimeInstance().format(f.lastModified())+
"t"+f.getPath());
if(!f.delete()){
System.err.println("Failed to remove "+f.getPath());
}
}
}
Util.getZxidFromName工具方法代碼
public static long getZxidFromName(String name, String prefix) {
long zxid = -1;
String nameParts[] = name.split("\.");
if (nameParts.length == 2 && nameParts[0].equals(prefix)) {
try {
zxid = Long.parseLong(nameParts[1], 16);
} catch (NumberFormatException e) {
}
}
return zxid;
}
